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The roles of assessment and testing within public 
employment and training programs supported by the Job Training 
Partnership Act of 1982 (JTPA) were e^-^inined through a review of the 
literature and two surveys of the JTPA system. In 1987, the Center 
for Remediation Design and the Center for Human Resources (CHS) at 
Brandeis University in Waltham (Hassachusetts) collaborated to 
interview managers in 150 service delivery areas (SDAs) that focused 
on basic skills testing. In 1988, CHS staff conducted additional 
interviews with about 100 SDAs and 12 states. Eliminating 
duplications and combining surveys yielded 205 surveys, with 176 
usable responses. Additional interviews were conducted with 
experienced state and local practitioi. s. A very high proportion 
(92%) of JTPA SDAs used formal testing. The JTPA system has 
recognized many of the shortcomings of standardized tests, especially 
language or cultural biases that are often relevant to minority 
applicants and clients. The trend toward basic skills education in 
jot! training is acknowledged in the purposes for which tests are 
used. Among JTPA testing professionals, there is a new appreciation 
of the utility of combining test results and other forms of 
assessment, with the use of objective tests to complement, and not 
replace, the professional judgments of well-trained staff. 
Statistical data are presented in three tables. A 47-ltem list of 
references are included. (SLD) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

*••******* ****n**mn***»*mn ************** 1,1,1, itniiiiit*mn***it**.in,i,t,itnit unit **** 



ERIC 



p 



Testing and Assessnsent 
in Publidy-^upported 
Job Training for the 
Disadvantaged 



July, 1989 



Prepared for: 

The National Commission 
on 

Testing and Public Policy 



Prepared by: 



Erik Payne Butler 
Lawrence Neil Bailis 



The Center for Human Resources 
Brandeis University 



PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER lERiC) " 



BEST COPY 




U 8 MPARrmtMT Of CDUCATKM* 

eOUCATlOJ^AL ft£SOUW:£S fNF0«MAT»ON 
CENTER tERtCl 

i<Th»& docum^i has tjeen rtpfodwcwJ m 
r Mtnof cnang©* ^a*^ t**'^ '"w*® »ft>p»oi'e 



mef^t do rw>t ft*c^M»f»*v »eDfe»e«^» otf»Ckat 
OC Rt poAitton or policy 



ACKNOWLEDGEMENTS: 



We want to thank Bernard Giffbrd and Linda Wing from the National Conimission on 
Testing and Public Poliqr for the opportunity to help us organize our thoughts on this 
subject. Sandwiched in among the press of intolerable schedules, this paper was 
something of a group effort Thanks to Sean Zielent^ch and Debra Axccl for the 

®^ organizing and canying out much of the telephone survey a hundred or 
so JTPA Service Deliveiy Areas, which required several timra that number of phone 
calls. Thanks to Alan Melchior and Andrew Hahn, our colleagues at Brandeis for 
assistance on questionnaire design and on the techniques for correlating the results 
from two surveys, and to Alan and to Susan Cuman for comments on the final draft. 
Susan Nealen made helpful contributions by drafting analyst of the survey results and 
suggested sections of this paper. TTie test listing and descriptions in Appendix A were 
prepared by the Center for Remediation Design and Brandeis for the National 
Commission on Employment Policy and borrowed directl>', and the brief "case study'* 
printed as Appendix B was prepared by a Brandeis graduate student, Margo Saddler. 
The rest of the prejudices, observations and conclusions, are ours. 



There's an old Army saying, *If it moves, 

salute it* Today, some reformers seem to be 
saying. *If It moves, test it* 

-Statement attribated to Gregory Aorig 
by Edward Fiske 

1.0 Introduction 

Americans have mixed feelings about tests. We fear taking them, we often don't 
like their results, but we keep using them. At times, seeking objectivity, we urge still 
more testing in our search for ways to overcome favoritism and discrimination in 
education, in employment, aiid a variety of other fields. At other times, we admit that 
the tests we use have biases and flaws, and our concern shifts to over-use and mis-use. 
And still we use them. Today the pendulum seems to be swinging towards our fearful 
side. Dean Bernard Gifford of the Graduate School of Education at the University of 
California at Berkeley, who chaired the Commission for which this paper is prepared, 
has lecently referred to a "stampede to more and more testing" and concluded that 
'^ere is increasing evidence of over-reliance on ^ test scores in making educational, 
training, and employment decisions." 
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As usual, there is something to be said for both sides of the argument. Most 
likely» we need to be able to find ways to define and then occupy the middle ground - 

on one hand, improving the validity of the t^ we use and making appropriate use of 
the information wc get torn them, and on the other hand, cuttii^ back upon or 
eliminating tests of questionable predictive validity. Tte question is how can this be 
done? 

The National Commission on Testing and Public Policy is ecamining this issue 
from a variety of different perspectives. This paper has been prepared to contribute to 
this broad-ranging review by summarizing the experience with assessment and testing in 
public employment and training programs. We begin with an overview of the Job 
Training Partnership Act of 1982 (JTPA), and of the ndes that assessment and testing 
are said to play within programs supported by JTPA. Subsequent sections blend 
subjective and objective evidence from thxee cycles of telephone interview and from ex- 
perience to discuss the actual uses of testing and assessment in JTPA The most 
significant current trend in employment training - an emerging 1)asic skills" movement - 
receives separate treatment from a testing and assessment point of Acw. The final 
section draws the implications of these experiences both for employment and training 
programs and for broader concerns of pubb'c poh'cy. 



2.0 The Approach Taken bv this Paper 

The perspectives underlying this paper draw primarily htm nearly forty 
cumulative years (we are isQ people, of on?y middling years) of personal experience 
with public and private employment programs., indudiqg JTPA and its predecessor 
employment and training prc^rams (maiiy of which were fiimled by the Cbmprehensivc 
Employment and Training Act - CETA. the Manpower Development and Training 
Act-MDTA, and the Economic Opportunity Act-EOA). However, it is also based on 
a review of the leading studies on JTPA implementation and two surveys of the JTPA 
system. 

In 1987, the Center for Remediation Design and the Center for Human 
Resources at Brandeis University collaborated to interview managers in 150 "Service 
DeUveiy Areas" (SDAs) that focused on basic skills testing. In 1988, Center for 
Human Resources staff undertook interviews of nearly 100 SDAs and tweWe states to 
gather data on which to base this paper. We also conducted a series of free-ranging 
interviews with a sampling of experienced state and local practitioners. 

The 1987 questionnaire was conducted with people chosen by a probability 
sample consisting of every third SDA on an alphabetized list, and can thus be 
generalized to all SDAs with a reasonable degree of certainty.' The interviews that 
were conducted during 1988 were designed to ascertain the extent of use of certain 
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practices in the field, and to sdicit the opinions of local and state ofiBcials about their 
evolution and meaning. Uiven this focus on informed opinion, the survey utilized a 
purposive sample of about 100 SDAs, including those whose leadership was known to 
us as well as a random sample; given this sampling strategy, one cannot assume that 
the sample is representative should not extrapolate the proportions in our findings to 
all SDAs. Moreover, parts of the 1988 survey were open-ended, the questions 
searching more fat ibf. ^nirrent opinions more than proof of numbers, and the analysis 
of the results has been cmfy crudely statistical. 

In several places, we have reanalyzed and incorporated the results of the 1987 
survey with the findings of the 1988 study. While the IS^ survey was focused 
specifically on basic skills testing, many of the responses yielded data which related 
directly to the subject of this paper. 

At times, we have taken some license and combined the results on similar items 
in the two surveys, eliminating the 47 interviews from the 1987 survey which duph'cated 
the SDA's among the 100 surveyed in 1988. 

But despite all of this statistical manipulation, both the conchvions drawn from 
these surveys and conversations, and the implications that are drawn for the broader 
frame of public policies in testing are our own. As noted earlier, this is primarily a 
personal review, written by two people who have worked in or with CETA at the 
federal and local levels, participated in the planning for JTPA, and watched JTPA's 
implementation dosefy and performed evaluation and policy research during both eras. 
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Our perspective is both supported and tempere^l by the work of our own organization, 
the Center for Human Resources at Brandeis University, with assessment-driven and 
competency-based employment programs for Uie disadvantaged clients, and by the \mTk 
we have done on this issue with the Department of Labor* the National Governors 
Association, the National Gommisaon on Employment Policy, and other groups. 

3*0 Overview of the Job Training Partnership Act 

When the Job Training Partneiiiiip Act was passed with some fanfare in 1982, it 
culminated a considerable bi-partisan policy sahmge job. The new federal 
Administration exhibited little real interest in job training for the disadvamaged, and 
based its skepticism on the well-pubbdzed difBculties of the Comprehensive 
Employment and Training Act CETA had peaked in annual budget at over $12 
Billion in 1979, and had ~ for reasons principally of bad management and unlucky 
public relations ahnost from its beginning in 1974 ~ become a fevorite whipping boy of 
critics of "social programs". 

The new Administration, looking for opportunities to cut programs from the 
budget, first drastically cut CETA's budget, then opposed its renewal when it expired in 
1981. However, there was clear support for doing something in emplpycicnt and 
training to replace CETA among both parties, in both branches of Congress. Thus, by 
mid 1982 Administration support began to appear for a bipartisan formulation known 
as the Job Training Partnership Act 
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The elements of this fonnulation - reflected in the bill which was finally passed 
signed in October, 1982 - included elimination of public service employment and most 
forms of subsidized work experience, the elevation of the role of the private sector 
through Private Industiy Councils, increased authority for the states, and a variety of 
measures designed to promote more rigorous, less eatpcnsiveiy administered, results- 
oriented management. 

Coupling these elements together with a reduced biulget request (less than $4 
biUion) made it possible to sell JTPA as a lean, mean, training-oriented, anti-make- 
work, pro-business, bottom-line driven initiative in the spirit of public-private 
partnership. This was not ''son of CETA", but a more distant relative. And so it has 
been in implementation. 

Despite predictable inertia at all levels, from the VS. Department of Labor 
through local Service Delivery Areas, JTPA - now in its seventh year, but only fifth of 
full-scale implementation - has developed into quite a diiferent employment and 
training system from its predecessor. States have become major players, often at the 
expense of local governments, which were firmly in charge during CETA The federal 
Department of Labor spent the first several years of JTPA's existence avoiding any 
meaningful programmatic role at all, short of toting up results, and most states were 
perfectly willing to pick up the slack. While the organized private sector is hardly in 
the driver's seat envisaged by some, certainty many more private employers are 
involved in JTPA than were during CETA Aside from the drastic reduction in budget, 



which has had the most dramatic impact, the institutional and political changes which 
have followed are, in some ways, the more noticeable. 

Quieter, less noticed stiU, but in some ways even more significant, are the 
changes in program and management practices ^cb have resulted from JTTA*s 
provisions calling for more outcome-orimted accountal^ty - notab^ the requirement 
that Governors monitor the performance of SDAs against certain outcome standards. 
CETA was heavily procedural, that is focused on participatory requirements for 
"significant population segments", targeting devices, and slot levels - and aknost before 
it got started it got burdened additionally with such confusing priorities as *^ublic 
service employment for countercyclical economic stimulation", an administrative and 
public relations disaster. 

JITA, on the other hand, has been ahnost fanatically outcome- driven, and both 
Administration and Omgress have, remembering CETA, strongly resisted "confusing" 
amendments even when technical and even programmatic adjustments were badly 
needed. 

The most significant outcome-oriented practice has been the development of 
formal national, state, and locally-admimstered systems of outcome measures, and 
standards for aggregated program achievement This so-called •^)crformance standard" 
system ~ driven by a nationally-derived set of outcome numbers against which the 
performance of local administrative cnt«*Ues, called Service Delivny Areas, or SDAs, 
(and, in turn, states) are measured, have become the basis for judgement about the 
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effectiveness of local programs, and, when they are aggregated, about the system as a 
whole. This rigorously-applied set of measures and standards has had the effect of 
turning the system on its head to comply. The measures have recently been expanded, 
and there have been annual adjustments in the standards, but in general there has 
been great ronsistenGy over the several years of JTPA, providing a good basis of 
ei^iience with outcome measures for a national program. 

TWo other practices that have evolved over the several years of full 
implementation of JTPA have either hardened or mitigated the impact of the 
performance standard system, and have created a new environment for assessment and 
toting practices in the field. The first is a set of approaches to "performance-based 
contracting** which, while not mandated ~ or even mentioned - by JTPA, has quite 
literally swept the system in the last three years. The second practice, authorized by 
JTPA, but little emphasized by the performance standard system in the first few years, 
is the development of competency measures for program outcomes. To some extent, 
the movement in JTPA towards competency measures, and in turn to competency- 
oriented training, though specifically authorized only for youth, has been a reaction 
against the existing performance standard system, which has emphasized Job placement 
over other outcomes. 

The drive for measurement and accountability in JTPA has bad a gradual but 
over time dramatic effect on the system's collective thinking about program services, 
and in turn about the uses of both informal assessment and formal testing. As 
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statewide systems and local programs are called upon to measure and document their 
aggr^gftK accomplishments, program managers are in turn pressed to better measure 
mdiv^dtiftl accomplishments. 

As we will suggest later in this paper, the actual uses of assessment and testing 
results vary greatly from dty to dty and even state to state, but the ectent of use of 
these techniques has grown considerably over the past five years. Though it is difBcult 
to document on a large scale, most of the people to whom we talked this past summer 
confirm our own impressions: that whereas ten years ago mc»t assessment in federally 
funded employment and training programs was done in a structured intake interview, 
much more reliance is now placed on results of formal testing. Thus, while the 
PrffPQytipn of local entities which claim to do some form of assessment at program 
entry has remained relatively constant, the ma between personal assessment and formal 
testing has changed significantly. 

4.0 gcQpe and lartent of Testing and Assessment Use in JTPA 

While there are "set asides" for the economically or educationally disadvantaged 
in various other publicly supported trpining and education programs, the Job Training 
Partnership Act has been the single largest program dedicated to providing job training 
services to this population. Fully operational in October 1983, it has been funded on a 
program year basis running from July 1st through June 30 each year. Total 
appropriations have ranged firom $ 3.3 billion to $ 3.9 billion over the ;ast several 
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years. JTPA training services are provided to displaced workers and to economically 
disadvantaged youth and adults. 

On average, just over 1,000,000 different individuals have received training each 
year in the primaiy training programs of JTPA (Titie II-A), with just fewer than half of 
them beii;g 21 years old and younger. In addition, appnsimatefy 700^000 additional 
youQg 2:«ople (under age 22) participate in the laige, six - eight week long, summer 
jobs program (Title D-B). For purposes of our current discussion, the reader should 
understand that a vexy high proportion of the million plus persons participating in 
training are tested or assessed in some furmal way. A much smaller, but growing, 
proportion of young people participating in the summer jobs program are also tested 
or assessed formally. 

Exhibit 1 on the following page combines the results of our 19S7 and 1988 
suiveys on testing and assessment to indicate the overaO extent of the use of testing 
and assessment methodologies in JTPA As is made clear by the exhibit, the vast 
majority of SDAs use at least some form of assessment, and roughly two-thirds of them 
combine intake interviews with formal tests. (These proportions apply principally to 
the main JTPA Title HA training programs.) 
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EKfaibit 1 

ES3£M OF USE OF TESTING AND ASSESSMENT IN JTPA*. 



RESPONSE 

Use £|ix method of assessment 
Use intake interview only 
Use fonnai test only 
Use both in combination/series 



£ PROPORTION 

164 93^% 

32 1S^% 

24 13.6% 

120 6&2% 



Combined results of 1987 and 1983 telephone surveys of 206 (non-dupUcates) 
Service DcKvcry Areas (150 in August 1987; 105 in July 1988, less 47 
duplications) usable responses on these questions: 176. 



Exhibit 2 reflects the decisions that local practitioners are making about which 
programs arc appropriate for formal testing methodologies. Roughly three-quarters of 
the respondents used formal tests for their adult classroom training programs, including 
about half of the SDAs who report using such tests for call cJassroom training. (As 
indicated in the note, these are proportions of the total of just over 80% of 
respondents who report using formal testing either by itself or in combination with 
informal techniques in an intake interview.) 
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Exhibit 2 

DIFFERENT PROGRAM TYPES IN WHICH FORNfAL TESTS ARE USED/ 




29 



20.1^0 




42 



292% 



Use testing for adult and youth 



73 



50.7% 



classroom traini ng participants 
only 



* Combined results of 1987 an ! 1988 Brandeis suiveys (less duplicated SDAs). 

Usable responses: 144. Proportions are calculated as percentage of respondents 
who report using tests sIqqs or in combination with other methods). 

In summary, a very high proportion of JTPA SDA*s are using formal testing, and 
nearly eveiy one uses some organized method of assessment or testing. As a result, 
nearly a million economically disadvantaged JTPA participants are participating in seme 
organized form of assessment, usually through formal testinfr ut program enrollment, 
often again subsequent to program participation. 
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5.0 The Purposes for which Testing is used in iT?A 

The 60O-pius Sendee Delivery Areas in JTPA have used standardized testing for 
at least five distinct purposes: (a) screening eligible applicants in or out of the 
program; (b) career planniQg, matching partidpams with job openings and/or assisting 
with job de^ dopment; (c) «Mg^gn«ng people ^ dififerenr components of the program; (d) 
monitoring progress of individuals; and (e) assessing aggregate and individual program 
impact Exhibit 3 suggests the proportion of SDAs using testing for each function. 
Each application is then discussed in the p&gt& which follow. 



Exhibit 3 

REPORTED PRINCIPAL PURPOSES OF ASSESSMENT AND TESTING' 



Program Intake (screening in or out) 29 453% 

Career Planning/Employability 56 87J% 
Development Planning/Job 
Development 

Assigning People to Dififerem 57 89.1% 
Program Components 

Monitoring Client Progress 32 50.0% 

Assessing Program Impact 39 60.9% 



* Source: Telephone survey conducted by Brandeis University staff in the summer of 
1988 with 64 usable responses on this item. (Note: Since th^ questions were not 
asked in the 1987 survey, we present data oni^ from the later survey.) 
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5.1 Screening Eligible Applicants Into or Out of the Pi 



Each job tratniog plan shall provide 
employment and training opportunities to those 
who can benefit from, tad who are most in need 
of, such opportunities. 

Job Training Partnership Act (PL 97-300. 
Sec 14 (c)) 

JTPA appropriatioits have never been sufficient to meet the needs of more than 
a small fraction of those who meet the income eligibility requirements for participation 
in the program, leaving SDAs to lefy upon the only the most general guidance in their 
efforts to decide whom to serve. SandeU and Rupp (1988), suggest that the intent of 
the JTPA legislation was to provide maximum local flexibility in participant selection. 
Maay SDAs have chosen to use objective tests and other assessment techniques as 
tools to help to decide which eligibles will actually get in the door, often by setting 
minimum literacy standards and using standardized testing to determine which clients 
meet the standards. 

Ostensibly, these tests are used to determine which eligible applicants can be 
helped the most by the program. As one survey respondent put it, "We have too little 
money; we have to target our pittance on people who can really benefit from it. 
Testing arms us with better information about which clients can succeed, so we depend 
on it heavily.** (Brandeis interview, 1988). 
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The "targeting" argument for testing is a double^ged sword. Not surprisingly, 
it is often alleged that tests are used for perverse purposes, especially "creaming", le., 
favoring easier-to^rve clients in lieu of those in greatest need among the eligible 
population (surely a tempting practice for prpgrams which must meet numerical 
standards for job placement and cost). 

Most oi the respondents to our surv^ of SDAs (say that th^) Jo not believe 
that the assessment and testing that th^ do at program intake contributes to creaming. 
However, there were wide variations among programs in the proportion who thought 
that assessment and testing M or jniglU have an influence, ranging from 19.2% for the 
JTPA dislocated worker programs to 47.1% for adult training programs. (Brandeis 
Interviews, 1988) 

Given the fact that "creamiug" is a dirty word in the JTPA lexicon, it seems 
likely that the reported estimates of this effect are underestimates, and we conclude 
that at least some of the assessment and testing approaches used in JTPA ^ tend to 
have "screening out" effects, especially in programs most held accountable to stringent 
cost and placement standards. Unfortunately, aside from our own opinions and those 
of the people we interviewed, the real extent of this practice is impossible to gauge 
because there are no objectively reliable data. 

Several points seem important to a fiiU understanding of the role of standardized 
testing in the screening process for JTPA ehgible apph'cants. The first is that it is not 
which tests are used, but the aay any test is used. For example, if an SDA imposes 
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minimum literacy thresholds for acceptance into the overall program or speinfic training 
courses, then the application of test data has the effect of redudng training 
opportunity. And it appears clear that that phenomenon occure with some frequency. 

Tht impact of these tests obviously depends what is done with them. At times, 
those who £all below certain standards are simply denied entrance to the program. But 
our survey also revealed instances in which those who &B below those threshold are 
referred to remediation programs run fay other agenc^ or schools or colleges. One 
SDA described a practice which applicants who are within one and a haif grade 
levels of the minimum entrance requirements for reading remain with JTPA for a brief 
remediation program, while those who are further behind are referred out to other 
agencies. Since there is evidently much attrition during or after this referral (some 
people simply never return), this process amounts to a negath« screen for those with 
lower educational attainment (Brandeis interviews, 1988) 

But strikingly, as masy SDAs seem to be using test scores to "screen people in" 
as Sticht (1980) would use the phrase, rather than the opposite. Several SDAs in our 
survey told us about efforts to eliminate apph'cants whose test scores were too high 
since they felt that they had less need for JTPA summer or year round programs. 

To be sure, tests can also have other unintended results. For example, there is 
widespread t^timoi^ to support the belief that the mere existence of objective tests at 
program intake screens out people who are unwilling to go through the testing process. 
Several SDA officials claim diat tests discourage people from enrollment because they 
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make the process long and cmnbersome. Even more importantly, managers perceive 
that large numbers of potential JTPA participants are reluctant to be tested at all, 
apparently bearing apprehensions that come torn a lifetime of (toing poorly on 
standardized tests.^ 

^•2 Career Development and Planning 

Most of the SDAs we interviewed told us that they use objective tests as part of 
their efforts to determine which services a program participant needs and to plan the 
kinds of outcomes for that participant that are both desirable and likely. One official 
tokl us, "the use of standardized tests provides us a more complete picture of the 
participant than is otherwise possible. We aui tell which program he or she needs, 
and ass^ them properly". (Brandeis Interview, 1988) 

Most of the SDA people we interviewed thought that the most effective 
assessments employ a combination of standardized tests and more subjective data, 
usually from intake interviews. One SDA ofiBcial told us that his assessmcr t staff start 
with a client's own expressed desires, and then "If the person's interests aren't 
compatible with the test results, we suggest, an alternative path. We couldn't do that 
with an interview alone or a test alone: we need both". (Brandeis Interview, 1988) 



^ iiwc WW to •cbool or hive tea out ofietool i kmg time 

ZS^^tlS J^J"i£S?!k'?**' Uktof. AD At»ku SDA offidtl toU n that U»e kind, peiplc 

IS^l2l?tte^^r*^ White thii i»«y te HtenOy t»e to thtt put of the amouy, it h i»(fc»N«tiy BpnwO^^ 



throughout the fyuem 
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On the other hand, many employment and training professionals are skeptical 
about the ability of sm standardized test to realistically ass^ job readiness and other 
key elements that go into an employability development plan. This has led some 
practitioners to tiy some awkward and expensive alternatives to testing. A number of 
the welfare enq>Iqyment prc^rams of the past decade have, for erample, opted to put 
all paitidpants throu^ trainiiig in job seardiiqg as an initial screen rather than use 
c^jective tests. While this hands-on noethod may work better*than a test to detenni&e 
which clients will need additional skills training, it does so at a relatively high emotional 
cost to the client who Sails to get a job (we are, after all, going through this program 
as 2§sg§smefll)> and at a high real cost to the program, since the training will have to 
be repeated for many clients when they are ready to it Yet this is an understandable 
impulse, in a field in which there is considerable mismatch between information needed 
for responsible training development and what is available from most standardized 
testing. Nevertheless, it violates the cardinal rule: "assess only for what you need to 
know, and use all that you get**. 

Assessment issues in JTPA received extensive attention during 1988 in a series 
of two-day training institutes on summer enrichment training institutes (SETIs) for state 
and local practitionen that were conducted by our Center for Human Resources here 
at Brandeis, as well as in a series of two-day "institutes" on basic skills assessments 
conducted by Brandeis in conjunction with the Center for Remediation design. Despite 
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the £Kt that more than 1200 practitioners were involved, some real consensus emerged 

from these discussions. They confirm the impressions drawn from our surveys: 

o There is no one current t^t that covers all of assessment needed in youth 
employment and training jwo^fams; 

o Forma], objective tests may be nec^saiy; but ttey are not a sufiBdent way to 
assess skiQs. Interviews and documents such as application forms and writing 
samples can provide good initial assessment information, and are needed in 
ahnost every case. 

o The utaity of assessments in career planning, developmem of employability 

development plans, and assigning participants to various educational and training 
components is enhanced when the t^ts are made understandable and 
meaningful to clients, t.e^ when trainees know wl^ they are being tested, what 
their score are, what they mean, and how tiicy relate to program and curriculum 
goals. 

o Information reeds to be gathered from all available sources. For example, an 
effective assessment system for youth may combine school-derived data from 
records and with employment and training results into a comprehensive ^tem 
which can be used both by schools and training agencies. 



If anything, we expect to see even more emphasis on the uses of tests for this 

purpose in the JTPA system in the future. A ttiirty-eight member national advisoiy 

committee to the Department of Labor stressed the need for an enhanced role for 

assessment in its final report, issued in March, 1989. Three of its headlined 

recommendations for the field are relevant to this d^cussion: 

Eveiy JTPA program should provide a diagnostic assessment and 
assignment process and a mix of sendees individualized to address participant 
needs». 

Only after participant's work history, Job and educational skills, interests, 
health, motivation and life circumstances are reviewed and documented, can 
they be used to establish the services individuals should receive and their 
obligations while receiving these services.... 
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States and local prc^^rams need to pay greater attention to the criteria by 
which appHcants are assigned to services and should collaborate on developing 
assessment tools that serve similar clients with multiple prpgram needs... 

S3 Matching Services to Client Needs 

JTPA programs offer a wide variety of services to participants - ranging from 
short-term efiiorts to help people team to look tot Jobs, through short and kHig term 
vocational skills training in classioom settings and on-the-job training efiforts. 
Standardized tests and related assessment procedures are often used to help decide 
which services are most appropriate for each client (This use of tests is closely related 
to the above-described use cf t^ts to develop career plans.) 

For the most part, SDA o£Bcials described growing reliance on the use of 
standardized tests for this purpose. One of them told us that a "complete battery of 
testing and assessment instruments" was necessaiy in order to allow program staff to 
properly match the cb'ent with the applicable training programs, and another noted that 
"academic and aptit jde testing" has led to a higher proportion of those entering 
classroom training programs actually completing them. She was reluctant to conclude 
that this was due to higher entry standards, arguing instead that it allowed "better 
matcbiQg" (Brandeis Interview, 1988). 

Another SDA offidal told us that standardized tests were used to establish 
thresholds for the reading abflity necessaiy to succeed in specific skills training 
programs: "Some clients simpfy canH handle the reading requirements for some of our 
programs; the tests help us make sure that none is put into a position where he or she 
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can't succeed- (Brandeis Interview, 19^). When properly done, this kind of screening 
is greatly valued by the agendes that have subcontracts with SDAs to actual^ deliver 
the training. 

Several SDA respondents cited a practical implementation dilemma, which 
characterized most large and middle-sized diy systems. Simply put, the question is: 
who does the assessment in a subcontract-based system? TraditionaUy, when 
assessment was more informal, and part of an intake interview, the agencies who 
actually delivered services performed the assessment As one person put it, This 
system meant that the assessment results were always skewed in the direction of 
services that agency provided itself. So you got a situation where diem A appUcs for 
job training at Agency X. Agency X offers clerical training for girls and automotive 
training for boys. So, practical^ speaking, no matter what the assessment really says, 
the message is: 'if you're male, you are offered automotive training - whether you need 
it or not*" (Brandeis Interview, 1988). 

As more formal testing is introduced, there has been growing tension between 
centralizing assessment to use it for referring to appropriate subcontract agendas and 
the wish by subcontractors to do their own testing, thereby, assuring themselves of 
appropriate enrolled Informal bureaucratic considerati(ms sudi as these can have a 
larger role than the specific namre of a test or the stated polides about the use of test 
results in frustrating attempts to use cHent information to match services to individual 
needs. 
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5.4 Monitoring and Measuring Propess of Partjcinants in TYaininp Programs 
While most testing in employment and training is done at intake, and applied to 
the purposes described earlier in this paper, there is a growing body of operience with 
using both formal and informal means to asi^ss in-program inogress of individuals* 
The term "benchmaridng" has come into ^mmon use on job tiainiog to describe the 
process by which interim prog re ss points are klentified and some fonn of assessment - 
usually informal, only rarefy formal testing - is used to determine whether a participant 
has reached a certain progress point, or benchmark. Two systems, however are 
increasingly being used in the field for this purpose. The Oomprehensive Competencies 
Program, developed by Robert Taggart, the former Administrator of the Office of 
Youth Programs in the Department of Labor, integrates pie- and post-tests and 
benchmark measures into an elaborate computer-managed basic skills instructional 
program now being used in over 300 local programs. OOP's various unit and level tests 
provide ongoing information on student progress, and is actually used in most programs 
more for instruction than for assessment 

A second system, the Comprehensive Student Assessment System (CASAS), also 
offers a system of assessment and benchmarking to show progress of learning. CASAS 
employs criterion-referenced tests, drawn from an extensive pool of test items designed 
for the pur|X)se, to measure specific employability skilled, to locate students and 
measure their progress along a continuum of basic educational skills. Through the 
work of the so-called "project of the states", developed by the Center for Remediation 
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Design, as well as through work by Brandeis and others, CASAS is now developing 
customized assessment and curriculum management systems for JTPA practiUoners in 
more than fourteen states. 

Our interviews iast summer revealed extensive use of interim measures, but a 
surprisii^ description of their purposes. Beyond OCP and CASAS, only a smattering 
Qi programs appear to use this approach in oonjunctfon with formal testing mechanisms 
in order to serve clients better. Many local SDA administrators do, however, report 
using interim pn^ress measures of some sort as part of their contracting system, 
enabling them to make progress payments to subcontractors. 

This may be because program managers are relatively well-satisfied with what 
they have. Several of them, in our interviews could not resist the nostrum, "if it ain*t 
broke, don't fix it" in response to questions about whether they were considering 
inttodudng more ambitious in-program measurement FuBy 82% were convinced that 
what they presently did (typically pre- and post-program assessment adequately meet 
the needs of clients and 87% reported that they were more interested in the kinds of 
interim measures which met their own management information needs about contractor 
performance. (Brandeis surveys, 1988). 

The slow growth of the practice of "bendmiarking" offers insight which confirms 
our instincts about the use of testing in JTPA. It appears that the willingness to look 
for interim measures of client accomplishment has been spurred in the past three or so 
years by the demands to make progress payments to subcontractors operating on 
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peifonnance contracts than b;^ educattonai concerns for measuring competency- 
development prepress (Brandeis Interview, 1988). But wl^ the movement to pre- and 
post tests? It appears to be bom of the same cause. Simply put, the JTPA 
regulations require it, and SDAs are (d>Iiged to employ the practice to determine their 
contractor payments. 

This has been an interesting development, revealed most clearly for us in the 
interviews of the past two years. It is worth setting some context here, to better 
understand these developments. Essentially, there are two trends, which began as 
program innovation, and have evened into managem«%nt practice, with their actual 
program service purposes taking a back seat Let us discuss these two issues - interim 
progress measurement or "benchmarking**, and end-of-prpgram competency 
measurements ~ separately, for each has come to emptoy assessment and testing for 
different purposes and in a different way. 

The development of l^enchmarking** began in the Job Corps programs of the 
late 1970's and hit its peak with the Comprehensive Youth Employment Program 
demonstration sponsored by the VS. Department of Labor from 1979 to 1981. 
Borrowing heavily from the competency based education movement, this set of more 
than a dozen federally supported pilot programs attempted to break down learning into 
bite size pieces, completion of each to be measured by some acceptable form of 
assessment, meanwhile linking training and work experience with education and 
supportive services. It was argued by those who helped to design the initic^ive (see for 
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example Fiala, 1982) that each individuBl panicipatmg in a job training program should 

have an employability development plan which was bufld around several elements: 

o Assessment ctf client interests aptitudes and vocational goals; 

o An EmptoyabOity Development Plan (EDP) in which assessment data and client 
counselor interviews determine types of services and training a client should 
receive; 

o Instruction based on the service needs specified in the EDP; 

o Reassessment (measurement) to determine whether and to vvbat extent the client 
has successfully completed a particular activity, and review of the EDP, and 
replanning; 

o Documentation based on a formal system to collect, ana^ and report accurate 
data on each individual dient; and 

o Evaluation of client and program data to determine both program and individual 
accomplishment. 

The idea, essentially, was to do for disadvantaged partidpants in job training 
programs what the Individual Educational Plan (lEP) and Individual Written 
Rehabilitation Plan (IWRP) have arguably done for special needs students and 
Americans with disabilities respectively. Each participant in publicly supported job 
Uainii^ would have, by this scenario, a vocational outcome objective (i.e., a job) and 
would be working on an indhidually monitored plan to produce in that person specific 
competencies to enable them to obtain and hold that particular job. And all of it 
would be driven by the interplay among assessment, instruction, testing and replanning. 

While the residual effects of this philosophy are still to be found, notably in 
some adult classroom training programs developing employees for specific occupations, 
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this occupational^ specific movement has by and laige given way to a focus on more 
general^ usually educationa!, competency achievements by youth and adult participants 
alike. In the adult training system, the JTPA performance standards have focused on 
measuring outcomes such as job placement and cost and have thus tended to displace 
concern with occupattonalty spedfic benchmarking. However, the tide may be turning 
in this aica; the above-cited 1989 JTPA National Advisoiy Gommittee report has 
recently called for adding 'Intermediate competency measures" in wder to "more 
effectively support basic skills remediation for adults." 

Meanwhile in the youth "branch** of the system the discussion of competency 
based youth employment programming has shifted focus to finding means to measure 
gains in reading, math, and communication skills and in what the field refers to as "pre 
employment skills" and "work maturity". These shifts in focus have meant, practically 
speaking, a mu^ greater focus on pre-assessment and post-assessment as means of 
documenting skills resulting from program participation. Not nearly as much progress 
has been made towards a genuine "benchmarking" approach to interim assessment and 
program adjustment 

As this ^lift has occurred, assessment and testing methodologies have also 
changed. Whereas ten years ago there was a search for the best methods of assessing 
specific vocational competendes, there ^ now a much greater focus in the field on 
kx)king for the best possible tests of reading and math competency. While some 
programs, especially those serving high school dropouts and the most disadvantaged 
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adults, use these mstnuneiits for 1)enchmarkiiig% applying them to interim 
measurement, most do not Instead, most program administrators (more than 80% of 
those we interviewed in 1988) report that they use more informal means of assessment, 
such as interviews and reports of counselling more instructional stafi, to measure in- 
program gains and rely upon standardized tests or other formal assessment for the 
beginning and the end. 

5.5 Assessinf Program Impact and Documenting System Progress 
Finally, most JTPA SDAs find themsehres aggregating data from pre and post 
program assessment and testing in order to develop estimates of program impact 
While nearfy four out of eveiy five SDAs in our survey indicated that they felt 
comfortable with the combination of pre-tests and post-tests for participant assessment 
many echoed the reservations of one SDA director who said. The problem with these 
reading and math numbers is that people actually believe them. And because the only 
other numbers we have are job placement and salaiy numbers, th^ think that teaching 
reading and math is all we do and they start judging us the way they would a school. 
We do so much more, and our clients start so far back, that it's a bit of a political 
piwVle. The dOemma as I see it is that we absolutely need to use these tests to do our 
job. Only a small part of our job is teaching reading and math, yet that's the only part 
for which we have bebevable measures." We need another kind of instrument. 
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designed by or with practitioners, to use for these purposes^ (Brandeis Interview, 
1988). 

There ^ enormous emphasis in the field on measuring program effectiveness. 
The system of performance standards, and the practice of performance contracting 
have literally swept the JTPA system, driving nearly every SDA to organize its 
programs and its contracting procedures to meet numerical goals established by the 
federal Department of Labor. But the measures of accomplishment have focused on 
job placement, wage gains, and per-person costs almost exclusively, and hardly at all on 
in-program gains in skills or knowledge, (see Butler, 1988) 

But this may change. The increased emphasis on educational skills as part of job 
training is actually a central development in this field. While there may well be a 
baddash against "using the job training system as an altemath^ education system", as 
oxjie SDA manager put it, in our opinion the trend is irresistible. More and more 
SDAs are going to be offering or contracting for remedial education services, especially 
instruction in reading and math. Not only does this development have the implications 
cited above for system measurement and accountability, but dearly it has big 
implications for the actual content of instruction and services in the job training system, 
to si^ nothing of the present and potential uses of assessment testing. Accordingly, this 
phenomenon warrants a brief separate discussion. 
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6.0 A New Development; Education and Educational Testinp in Job Training 



".inadequate basic academic skills are 
intertwined with youth employment and with 
dropping out of tehooU out of wedlock 
parenting, welfare dependency* and decline in 
workforce productivity growth.* 

•<<3ordon Berlin and Andrew Sum, The 
Ford Foundation, 1988. 

"...Young people entering the workforce 
today need the ability to learn and to adapt to 
changes in the workplacctbe ability to learn 
will be the essential hallmark of the successful 
employee." 

-<^mmittee for Economic Development. 
New York City, 1985 

The job training community has bardfy been immune to the discussions whirling 
around changes in demography, poor educational preparation, and the crisis in literacy. 
Indeed, the emerging consensus about the relationship between literacy skills and 
economic productivity has found a home in the public policy debates surrounding the 
Job Training Partnership Act Not surprisingly, this consensus has begun to have an 
effect on the program services delivered under JTPA and on the way in which testing 
and assessment methodologies are used and applied. When the General Accounting 
OfSce surveyed JTPA SDAs in 1986, 57% reported offering basic skills for mediation 
in at least some of their programs. The telephone interviews conducted by Brandeis 
and the Center for Remediation Design in mid-1987, found that virtually eveiy SDA 
provided at least some educational remediation in their programs, 28% during summer 
programs only and nearly 70% in both summer and school year activities for young 
people. 
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The spread of basic skills instruction tbroughout the JTPA system has been 
spurred by the increasing^ powerful message (by now, oft repeated) that the ability to 
read, write and compute effecth^fy is an essential requirement fior employment 
Moreover, an unacceptabfy large proportion of the disadvantaged young people have 
fiailed to develop those skills in traditk>nal dassroom situaSimis^ both in training 
programs ami in schools. It is no coincidence that the movetnoit has also been 
spurred through re-stated congressional mandate in the fiorm d amendments to the 
legislation and armtwisting of DOL administrators. 

But in responding to this challenge max^ JTPA administrators and planners ere 
moving onto new ground. They are learning as they go, making decisions about 
program design, curriculum development, testing and assessment, selection of 
contractors and coordination with educational institutions to an unprecedented extent 

Having sa^d this, we must also note that the penetration of basic skills 
instruction into job training has been wide but it has not been deep. Whereas nearly 
eveiy SDA is offering some basic skills remediation, most are offering it for only a 
handful of youth and adults. Thus, while nearly a hundred percent of SDAs are 
offering some remediation, the GAO reported that in 1986 only 8% of summer 
participants received educational services. 

This is another area where favorable changes appear to be occurring. Thus by 
our 1987 survey the number had grown to 20%. Responding to the mandate of the 
US Department of Labor for the summer of 1988 the number of summer participants 
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who have received some basic skills educational remediation had grown to nearly half. 

A similar [»ttem appears to hold for the one million or so youth and adult 
participants in the main year-round JTPA training prc^ram. Both word of mouth 
reports and responses to our nmi-sdentific 1988 survey suggest that a much smaller 
share* certainly tess than 10%, are participating in education programs as part of their 
job training. Noi»thetess, it also seems dear that proportion is growinig. As one SDA 
respondent put it, '^u can't believe how much pressure we get from the private sector 
members of om Private Industry Council to teach people how to read. Our PIC is 
divided into two camps: one says train *em £Eist and get them into a job. The other 
one says if you haven't taught them how to read you haven't done anything and I don*t 
want to hire them." (Brandeis Interview, 1988) 

We will not here repeat the arguments for and against JTf A*s involvement in 
basic skills instruction or the prescriptions for better programming, which tempt us. 
Suffice it to say again that this is a major trend in this fiekl, one which bea/s watching 
closely in the next few years. This trend towards basic skills education in job training 
began in the youth program side of JTPA, finding its policy roots in the work of the 
Vice President's Task Force on Youth Employment in the waning years of the Carter 
Administration, and surviving the JTPA wars to re-emerge with the national push for 
improved education. It is pan of a larger shift, from job placement to job preparation . 
While that may seem a too-obvious distinction, we believe it is fair to observe that 
JTPA's first several yeais have emphasized placement over preparation, seen in its 
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emphasis on placement - driven performance standardSi short duration of training 
programs, and patterns of client selection. Gradually, the shift to preparation is seen 
m longer programs, more use of assessment to plan program services, and in the 
introduction ctf seivices designed to develop basic educational skills. 

Why is this important for testing and assessment policy? First, because it puts 
formal testing squarely in tkc middle of employment and training developments. Here 
is a set of tools (standardized tests) developed generally for sdtool based programs. 
Now practitioners in employment and training are being asked for information which 
can only be generated by standardized tests. Can they use the on^ they have? Can 
they adapts others? Do they need their own special test? As employment and training 
practitioners expand their use of testing in an effort to assess and document 
employment - related basic skills, they are beginning to see how complex that set of 
questions really is. Current trends suggest that new instruments will be needed. At the 
same time, as the use of testing expands, it highlights the need for better (any, in many 
cases) staff training in selection and use of assessment to assure they produce 
conjiistent and appropriate results. 

Nor is this a trivial set of developments, stuck away in some obscure program 
comer. There are potential^ lots of people invoh^ed. More than a million youth and 
adults in training programs are potential beneficiaries or victims of good applications of 
standardized tests and both formal and informal assessment by other means. And 
another 750,000 or so young people may be tested av part of their participation in 
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summer programs. As a consequence, what happens in JTPA with testing may teach 
us a great deal about the relevance of testii^ to educational practice with a set of 
disadvantaged clients. JTPA participants aie almost all ofiEkaalty poor, most are 
minority and urban and therefore likely to be educationally as well as economically 
disadvantaged. As this set of pn^rams shifts from job placement to more 
stratghtfoiward educatkm and training, it is likely that the use of testing in programs 
designed for thte population may yield fruitful insets both for education and for 
toting. 
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7.0 Use of Formal Testing Instmments in JTPA 

Dedsions about which fonns of assessment to use are made by more than 600 

relatively autonomous JTPA Service Delivery Areas. As a result, it is not surprising 

that there is a wide variation in the extent to which th^ vsc fiormal testing mstruments, 

and the specific instruments that they choose to employ. Hie 1987 survey provides 

some insights into the choices which were made at that time (and which are probably 

still being made today.)' Hi^ilights of the study include: 

o 92% of the respondents reported using at least one standardized test 

o Those SDAs that choose to use "off the shelf standardized tests have tended to 
utilize one or more of four tests: 

- 393% used the Tests of Adult Basic Education (TABE); 

- 22.7% used the (CAT) 

- 16.7% used the (WRAT) 

- 9.3% used the (ABLE) 

- No other test was used by more than 2.9% of the sample 

o Roughly one in fourteen SDAs (7.3%) developed their own tests for these 
purposes. 

o hi general, SDAs tended to supplement their standardized tests with other 
information, either outside sources such as the schools, or Judgments derived 
from the intake interview. 
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a reasonable degree of oemioiy. 
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Testing profwsionals within the JTPA system are bcconiing increasingly aware of 
shortcomings of standardized tests such as the above-dted ones as oppo^ to criterion- 
or competency-referenced tests such as the Comprehensive Adult Student Assessment 
System (CASAS). Agieein^ the JTPA Advisoiy Committee suiggpsted that the 
development of criterion-ieferenced tests for the fiehi ovght to be a h|gh priority. 

^•0 Conclusions about Assessment and Testing in ffp^ 

As noted above, the more than 600 Service Deliveiy Area agencies tt -t deliver 
JTPA services differ widely in the degree to which they use (Ajjective testing and the 
degree of sophistication that they use in interpreting the results. But taken as a whole, 
the JTPA system has moved a good distance in coming to understand the uses and 
misuses of standardized testing. The conclusions that can be drawn are four-fold: 
o When over-used, or mis-used, objective tests can be counterproductive. 

o When properly used, objective tests can be a useful complement to professional 
judgments of well trained sta£L 

o It is impossible for objective tests to be properly used without appropriate 
training for those who select and administer them as well as those who must 
interpret the results to make decisions. And not enough training is being done 
presently. 

o More can be done to develop and validate the kinds of tests that are feasible 
for agencies with limited budgets to use. 
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8.1 Shortcomings of Objective T«tts 

JTPA practitioneis report that tbey have become increasingly aware of the 

shortcomings of standardized tests such as the potential for language or cultural biases 

that are especially retevant to many of the applicants for and clients in their programs, 

While this chall^^ exists for all of JTPA's laigely poor population, the problems of 

Hispanic clients who are illiterate in English or Spanish or both have proven to be 

especially vexing to SDAs ^ want to serve them but have neither testing nor 

instructional materials in Spanish, and veiy few well-prepared staff to serve such cUents. 

(Brandeis interviews, 1988), though CASAS is developing an employment-related ESL 

assessment and curriculum management system for use in Jl'PA. 

The awareness of the limitations of testing extends beyond cultural bias issues to 

its general utility. This view is exempli0ed by other responses to our 1988 sunrey. 

Here is a smattering of comments: 

"Testing is only one aspect of a properly done assessment" 

Testing as we know it just does not adequately measure the variables 
that are most important to us in JTPA, like motivation and things that 
employers really care about - work ethic and reliabihty". 

The tests that are now available are too narrowly focused. Tbey can't 
look at the broader picture and don't capture the broader side of people". 

The tests that we are able to administer are not thorough enough to give 
a really good picture of the client and his or her needs and potentials. At best 
they are only a partial picture. Fm tots more impressed by the usefulness of a 
good intake interview than by the results of reading or vocational skills tests". 

There are too many intangibles in this business to expect to capture the . 
important stuff with tests. It is difBcult to categorize human t>eings with 
different devetopmental rates". 
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Tests can be useful, but other things can be even more important to 
counselors such as clients* interests and personal interactions with the client 
But most of the real decisions are made by other things - the number of slots 
available and/or funding left in specific training courses". 

At least a few respondents to our survey were even stronger in their skepticism. 
Five of those we interviewed seemed to agree with one of their colleagues who said, 
Tm tiying with all my power to resist doing ai^ testing at aH I think it's all gone too 
far, and I want to keep at least one aspect of my clients* lives free from 
measurement - Vm losing, but Fll go down swinging". (Brandeis interviews, 1988). 

^ Combming Objective Tests and Professional Judpement 
This awareness of both the value and the shortcomings of objective tests has 
been translated into a new appredation of the utility of combining tests results with 
other forms of assessment The JTPA experience has thus made it clear - to 
practitioners and to us - that formal testing should be seen as a worthwhile 
Pftmpl^m??^! to professional judgments of well trained staff; not a replacement for 
them. JTPA testing professionals have concluded that no single objective test or 
batteiy of tests can provide a perfectly valid measure of deficiencies in education 
and/or job readiness; given this situation it would be unreasonable to place total 
dependence on such measures. Indeed, it is hard to imagine that any other field would 
responsibly come to a far different conclusion. 
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These opinions are echoed in many of the comments ofifered by SDA 

respondents to jur 1988 survey: 

"I'ssts do a reasonable job of both screening and matching if they're 
combined vyith solid counsetiqg to bade up their interpretation^ 

"A combination of testing, school results, and interview data is helpful in 
detenninii^ which program clients go to and which services they gpt". 

"If 8 client is willing to go througlh a barrage of tests, more information 
will be availaUe that can contribute to more accurate service", (emphasis 
added) 

83 Resource Requirements 

In many cases, JTPA staff have tended to underestimate the amount of time and 

resources necessary to properly utilize objective tests. The SDA respondents to our 

1988 survey repeatedly expressed frustration over the feet that the tests that they 

wanted to use were too expensive, required more staff time than was available, 

required a better trained staff than was feasible, and were excessively compb'cated and 

time consuming. For example, we were told that: 

1 wish we would find an assessment tool to more accurately capture what 
the client can do in less time". 

and, 

"Explaining test results and what they really mean for a client's future can 
take days". 

This growing realism about the proper utilization of tests in the JTPA syUcm is 
a positive sigxk, but no easy answers to the challenges have emerged at a time when 
budgets for federally-funded programs are being squeezed increasingly tightly. There is 
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clearly a market for tests that can provide useful guidance with minimal resource 
requirements but little certainty about when or how it will be met 



8w4 Training 

There is a dawning awareness among JTPA Sendee Deiivexy Areas and their 
subcontractors that it is not enoiigh to have the best tests and sufficient staff to 
administer them. Staff must be well trained in the theories that underlie specific tests, 
the ways that they should be administered, and (perhaps most importantly) the kinds of 
conclusions that can (and cannot) be made in interpreting test results. This point was 
most succinctly made by a 1988 survey respondent who said: 

"Our staff people are not professional testers. They don't 
know enough to make fiill use of the tests that we do use. Ym 
worried that well go through a rote exercise to comply with 
someone's notion that testing is useful, and either not understand 
how to give the test or how to use what we are supposed to be 
learning from it". 

Training has also contributed to some of the insights about proper use and 
misuse of tests that we have already discussed. For example, several SDA ofiBcials 
agreed with the spirit of one of their colleagues' conunents: 

"As a result of training, our staff now place less reliance on raw testing 
data, and more on competency-based and interview-based assessments. This 
combination of fiictors results in better decisions as to who gets into the 
program and better referrals within the pr(^ram. It resuhs In better cHent 
decisions about what they can hope to do, and better provider decisions about 
what they can hope to accomplish with the clients.» 

"People in the system who are not educators or otherwise trained in 
testing want to document eveiything. They tend to push for too much 
standardization rather than using professional judgment.. 
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"The argument that testing can be helpful in running a JTPA program 
assumes and prays that administrators and instructors how to use assessment and 
are doing it properly". 

Finally SDA administrators are becoming increasing!^ sophisticated about the 

need for training in how they can assess the variety of available tests and decide which 

ones are most appropriate for their clients. As one of them tcdd us: 

"We are still stniggUog to develop appropriate tests and measurement 
devices. Tm not sure that there i& a good enough body of knowledge developed 
yet, though I think we're getting there". 

&5 Need for More Research. Development, and Validation 
While aware of potential shortcomings, most SDAs vath whom we spoke felt 
more or less satisfied with the tests that they were using, or at least unconvinced that 
there were any better alternatives. The epitome of this point of view is the SDA 
o£5cia] who said. The tests we use are okay~we think." 

Despite this point of view, it goes without saying that as the uses of assessment 
in JTPA are growing, more attention needs to be paid to efforts to develop, validate, 
and promote the utilization of improved assessment methodologies that are both valid 
and easy to employ and interpret As noted in the 1989 JTPA Advisory Committee 
report: 

The ongoing effort to develop and implement national assessment tools 
and standards should be supported with substantial technical assistance from the 
federal level 
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This does not mean that we can ever expect researchers to come up with a 

single test that aO JITA managers will be able to emplc^ for all purp(»es. As noted in 

Monis, Stnimp^ and Cuman (1988): 

There ^ not one best test Assessment is an (mgoing process and as such 
is as much an art as a sdcncc-no perfect or complete strategy exists. Many 
variaUes affect the test selection 8^>ect d the a ssessmen t process: the target 
groups, the participant outcomes expected, the amount and type oi existing 
assessment information available, and tlw amount of dollais available. What is 
best for the needs of one program and cfiem group may not be as effective for 
another. 

As is tiite in some many other aspects of employment and training, what we really 
need is research and development to help us understand what kinds of tests work best 
for whom under what circumstances. 

8.6 Limitatjons 

Lastly, while many JTPA practitioners are becoming more sophisticated about 
testing issues, most are still confused about the basic purposes of assessment and 
testing and its appropriate use. In part this is explained fay history ~ they are in the 
testing business because of requirements of regulation and because of the need to 
document performance of contractr and of local systems ~ Ve do pre and post testing 
because the regs require it to claim competency attainment in our performance 
standard reports'*-. likewise, Ve test summer kids because it's required, but the 
program is up and down so quick it never affects what we do with them. We don*t see 
the point of this mudi hassle for this little gain...". It is not, therefore, surprising that 
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often inappropiiate tests are used, or (maybe worse), that tests are given and little use 
is made of the results. 

At the same time, rapid introduction of practitioners to the world of testing has 
led to what may be an over-concern about the precision of tests themseKes. Too often 
the questions of Vhat is the best test" reflects a search ficn* an ehish« perfiect source of 
information. In this field, the real purpose of assessment is to help locate appropriate 
starting points for young people and adults being tested, and to serve as practical tools 
of measuring progress toward emplqyability. They need to be treated as practical tools 
rather than as arbiters of some objective truth. Ironically, used that way, tests also 
become better documentary devices for program progress when individual progress data 
are aggregated. Driven by the outcome motive alone, it may be that the real 
integration of information into client sorting, program planning, and progress mapping 
may be an elusive goal. Which would be sad, for these are likefy the most important 
uses of testing and assessment in this field. 
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9.0 Implications for Other Fieids 

Is the case of JTPA unique? We don't think so. As we see it, JTPA provides 
clear insists into the strengths and weaknesses of the use of standardized tests for 
assessment purposes in our society because the program encompasses both education* 
related and job-related criteria and measures. The ITPA lessons about the need to be 
alert to misuse of objective tests, using objective tests to complement-^d not replace- 
the professional judgments of wen-trained stafi; the need to develop tests that are vah'd 
and feasible to use, and the need to train staff to use tests properly are dearly relevant 
to all efforts to assess educational achievement and job readiness. And we wouldn't be 
surprised if their relevance extends far beyond that. 
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